With the proliferation of digital cameras in portable, ruggedized, and desktop devices, there is a related desire to use these cameras in new ways. Some of these ways work with faces that are within the camera's field of view. By recognizing and interpreting faces, many new functions are provided, such as identifying a known user, interpreting a user's expressions as commands or as other input, mapping a user's expressions to an avatar for a chat session, determining whether a user is paying attention to something in the user's field of view, and more. By recognizing a face, a computing system may be able to apply digital makeup, hairdos, hats, clothing, and other fashion looks. Two and three-dimensional face mesh generation may be performed as well as many other augmented imaging functions. These functions are useful in portable and wearable devices, such as smart phones, action cameras, and headsets and are also useful in notebook, desktop, gaming, and entertainment systems.
In many techniques, in order to recognize and interpret faces, first a face region is identified. This may be done for a single image or for a sequence of images in a video. The images may be two or three-dimensional. Once a system determines that a particular region may have a face, then the alignment of the face is determined. Face alignment is a technique to locate the positions of a set of semantical facial landmark points, such as ears, eyes, nose, mouth, chin and cheeks. These facial landmark points may then be used to determine the direction toward which the head is facing, to track eye movements, to recognize or distinguish two different faces and more. Face alignment is used in many computer vision systems including face recognition, facial feature tracking, and animation.
Face alignment is made difficult because of the differences between faces of different people and because of the differences caused when a single person changes expressions. Another difficulty comes when a person turns the head or when a head is partially blocked or occluded or blocked by another person or an object. Face alignment is also made difficult by variations in backgrounds behind a face and illumination around the face.